通过 Terraform 管理 Kafka Topic

Topic云消息队列 Kafka 版里对消息进行的一级归类,使用Kafka的第一步就是先为您的应用创建Topic。本文介绍如何通过Terraform提供的alicloud_alikafka_topic资源创建、删除及查询Topic。

说明

本教程所含示例代码支持一键运行,您可以直接运行代码。一键运行

前提条件

  • 由于阿里云账号(主账号)具有资源的所有权限,一旦发生泄露将面临重大风险。建议您使用RAM用户,并为该RAM用户创建AccessKey,具体操作方式请参见创建RAM用户创建AccessKey

  • 通过RAM授权,阿里云用户可以有效地管理其云资源访问权限,适应多用户协同工作的需求,并且能够按需为用户分配最小权限,避免权限过大导致的安全漏洞‌。具体操作方式请参见RAM用户授权

    {
        "Version": "1",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "vpc:DescribeVpcAttribute",
                    "vpc:DescribeRouteTableList",
                    "vpc:DescribeVSwitchAttributes",
                    "vpc:DeleteVpc",
                    "vpc:DeleteVSwitch",
                    "vpc:CreateVpc",
                    "vpc:CreateVSwitch"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": "bss:ModifyAgreementRecord",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "bss:DescribeOrderList",
                    "bss:DescribeOrderDetail",
                    "bss:PayOrder",
                    "bss:CancelOrder"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ecs:CreateSecurityGroup",
                    "ecs:ModifySecurityGroupPolicy",
                    "ecs:DescribeSecurityGroups",
                    "ecs:ListTagResources",
                    "ecs:DeleteSecurityGroup",
                    "ecs:DescribeSecurityGroupAttribute",
                    "ecs:AuthorizeSecurityGroup",
                    "ecs:RevokeSecurityGroup"
                ],
                "Resource": "*"
            },
            {
                "Action": "alikafka:*",
                "Resource": "*",
                "Effect": "Allow"
            },
            {
                "Action": "ram:CreateServiceLinkedRole",
                "Resource": "*",
                "Effect": "Allow",
                "Condition": {
                    "StringEquals": {
                        "ram:ServiceName": [
                            "connector.alikafka.aliyuncs.com",
                            "instanceencryption.alikafka.aliyuncs.com",
                            "alikafka.aliyuncs.com",
                            "etl.alikafka.aliyuncs.com"
                        ]
                    }
                }
            }
        ]
    }
  • 准备Terraform运行环境,您可以选择以下任一方式来使用Terraform。

    • Terraform Explorer中使用Terraform:阿里云提供了Terraform的在线运行环境,您无需安装Terraform,登录后即可在线使用和体验Terraform。适用于零成本、快速、便捷地体验和调试Terraform的场景。

    • Cloud Shell:阿里云Cloud Shell中预装了Terraform的组件,并已配置好身份凭证,您可直接在Cloud Shell中运行Terraform的命令。适用于低成本、快速、便捷地访问和使用Terraform的场景。

    • 在本地安装和配置Terraform:适用于网络连接较差或需要自定义开发环境的场景。

使用的资源

说明

本教程示例包含的部分资源会产生一定费用,请在不需要时及时进行释放或退订。

创建Topic

本示例以在华南1(深圳)地域下创建Topic为例,Topic名称为example-topic。Topic创建完成后不支持修改名称。

  1. 创建一个工作目录,并在该工作目录中创建名为main.tf的配置文件,然后将以下代码复制到main.tf中。

    • 创建前置资源。

      variable "region" {
        default = "cn-shenzhen"
      }
      
      variable "instance_name" {
        default = "alikafkaInstanceName"
      }
      
      variable "zone_id" {
        default = "cn-shenzhen-f"
      }
      
      provider "alicloud" {
        region = var.region
      }
      
      # 创建VPC
      resource "alicloud_vpc" "default" {
        vpc_name   = "alicloud"
        cidr_block = "172.16.0.0/16"
      }
      
      # 创建交换机
      resource "alicloud_vswitch" "default" {
        vpc_id     = alicloud_vpc.default.id
        cidr_block = "172.16.192.0/20"
        zone_id    = var.zone_id
      }
      
      # 创建安全组。
      resource "alicloud_security_group" "default" {
        vpc_id = alicloud_vpc.default.id
      }
      
      # 创建实例,磁盘类型为高效云盘,磁盘容量为500 GB,流量规格为alikafka.hw.2xlarge。
      # 部署实例。
      resource "alicloud_alikafka_instance" "default" {
        name           = var.instance_name
        partition_num  = 50
        disk_type      = 0
        disk_size      = 500
        deploy_type    = 5
        io_max_spec    = "alikafka.hw.2xlarge"
        vswitch_id     = alicloud_vswitch.default.id
        security_group = alicloud_security_group.default.id
      }
    • main.tf文件中增加resource "alicloud_alikafka_topic" "default" {}配置项。

      variable "partition_num" {
        default = "12"
      }
      
      variable "remark" {
        default = "kafka_topic_remark"
      }
      
      resource "alicloud_alikafka_topic" "default" {
        instance_id   = alicloud_alikafka_instance.default.id
        topic         = "example-topic"
        local_topic   = "false"
        compact_topic = "false"
        partition_num = var.partition_num
        remark        = var.remark
      }
  2. 执行以下命令,初始化Terraform运行环境。

    terraform init

    返回如下信息,表示Terraform初始化成功。

    Initializing the backend...
    
    Initializing provider plugins...
    - Checking for available provider plugins...
    - Downloading plugin for provider "alicloud" (hashicorp/alicloud) 1.90.1...
    ...
    
    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.
    
    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
  3. 创建执行计划,并预览变更。

    terraform plan
  4. 执行以下命令,创建资源。

    terraform apply

    在执行过程中,根据提示输入yes并按下Enter键,等待命令执行完成,若出现以下信息,则表示运行成功。

    alicloud_vpc.default: Refreshing state... [id=vpc-****]
    alicloud_security_group.default: Refreshing state... [id=sg-****]
    alicloud_vswitch.default: Refreshing state... [id=vsw-****]
    alicloud_alikafka_instance.default: Refreshing state... [id=alikafka_post-cn-****]
    
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
    following symbols:
      + create
    
    Terraform will perform the following actions:
    
      # alicloud_alikafka_topic.default will be created
      + resource "alicloud_alikafka_topic" "default" {
          + compact_topic = false
          + id            = (known after apply)
          + instance_id   = "alikafka_post-cn-****"
          + local_topic   = false
          + partition_num = 12
          + remark        = "kafka_topic_remark"
          + topic         = "example-topic"
        }
    
    Plan: 1 to add, 0 to change, 0 to destroy.
    
    Changes to Outputs:
      + partition_num = (known after apply)
    
    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.
    
      Enter a value: yes
    
    alicloud_alikafka_topic.default: Creating...
    alicloud_alikafka_topic.default: Creation complete after 8s [id=alikafka_post-cn-****:example-topic]
    
    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
  5. 验证结果。

    执行terraform show命令

    执行terraform show查看Kafka Topic信息。

    terraform show
    # alicloud_alikafka_topic.default:
    resource "alicloud_alikafka_topic" "default" {
        compact_topic = false
        id            = "alikafka_post-cn-****:example-topic"
        instance_id   = "alikafka_post-cn-****"
        local_topic   = false
        partition_num = 12
        remark        = "kafka_topic_remark"
        topic         = "example-topic"
    }

    登录云消息队列 Kafka 版控制台

    登录云消息队列 Kafka 版控制台,查看Kafka Topic信息。p874255.png

更新Topic

本示例以修改Topic的分区数和备注为例。

  1. 在上述main.tf文件中,修改variable "partition_num"variable "remark"default参数值,具体配置如下。

    variable "region" {
      default = "cn-shenzhen"
    }
    
    variable "instance_name" {
      default = "alikafkaInstanceName"
    }
    
    variable "zone_id" {
      default = "cn-shenzhen-f"
    }
    
    variable "partition_num" {
      default = "24"
    }
    
    variable "remark" {
      default = "updated_kafka_topic_remark"
    }
    
    provider "alicloud" {
      region = var.region
    }
    
    # 创建VPC
    resource "alicloud_vpc" "default" {
      vpc_name   = "alicloud"
      cidr_block = "172.16.0.0/16"
    }
    
    # 创建交换机
    resource "alicloud_vswitch" "default" {
      vpc_id     = alicloud_vpc.default.id
      cidr_block = "172.16.192.0/20"
      zone_id    = var.zone_id
    }
    
    # 创建安全组。
    resource "alicloud_security_group" "default" {
      vpc_id = alicloud_vpc.default.id
    }
    
    # 创建实例,磁盘类型为高效云盘,磁盘容量为500 GB,流量规格为alikafka.hw.2xlarge。
    # 部署实例。
    resource "alicloud_alikafka_instance" "default" {
      name           = var.instance_name
      partition_num  = 50
      disk_type      = 0
      disk_size      = 500
      deploy_type    = 5
      io_max_spec    = "alikafka.hw.2xlarge"
      vswitch_id     = alicloud_vswitch.default.id
      security_group = alicloud_security_group.default.id
    }
    
    resource "alicloud_alikafka_topic" "default" {
      instance_id   = alicloud_alikafka_instance.default.id
      topic         = "example-topic"
      local_topic   = "false"
      compact_topic = "false"
      partition_num = var.partition_num
      remark        = var.remark
    }
  2. 创建执行计划,并预览变更。

    terraform plan
  3. 执行以下命令,创建资源。

    terraform apply

    在执行过程中,根据提示输入yes并按下Enter键,等待命令执行完成,若出现以下信息,则表示运行成功。

    alicloud_vpc.default: Refreshing state... [id=vpc-****]
    alicloud_security_group.default: Refreshing state... [id=sg-****]
    alicloud_vswitch.default: Refreshing state... [id=vsw-****]
    alicloud_alikafka_instance.default: Refreshing state... [id=alikafka_post-cn-****]
    alicloud_alikafka_topic.default: Refreshing state... [id=alikafka_post-cn-****:example-topic]
    
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
    following symbols:
      ~ update in-place
    
    Terraform will perform the following actions:
    
      # alicloud_alikafka_topic.default will be updated in-place
      ~ resource "alicloud_alikafka_topic" "default" {
            id            = "alikafka_post-cn-****:example-topic"
          ~ partition_num = 12 -> 24
          ~ remark        = "kafka_topic_remark" -> "updated_kafka_topic_remark"
            tags          = {}
            # (4 unchanged attributes hidden)
        }
    
    Plan: 0 to add, 1 to change, 0 to destroy.
    
    Changes to Outputs:
      ~ partition_num = "12" -> "24"
      ~ remark        = "kafka_topic_remark" -> "updated_kafka_topic_remark"
    
    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.
    
      Enter a value: yes
    
    alicloud_alikafka_topic.default: Modifying... [id=alikafka_post-cn-****:example-topic]
    alicloud_alikafka_topic.default: Modifications complete after 2s [id=alikafka_post-cn-****:example-topic]
    
    Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
  4. 验证结果。

    执行terraform show命令

    执行terraform show查看修改后的Kafka Topic信息。

    terraform show
    # alicloud_alikafka_topic.default:
    resource "alicloud_alikafka_topic" "default" {
        compact_topic = false
        id            = "alikafka_post-cn-****:example-topic"
        instance_id   = "alikafka_post-cn-****"
        local_topic   = false
        partition_num = 24
        remark        = "updated_kafka_topic_remark"
        tags          = {}
        topic         = "example-topic"
    }

    登录云消息队列 Kafka 版控制台

    登录云消息队列 Kafka 版控制台,查看Kafka Topic信息。image

查询Topic

  1. 在上述main.tf文件中,增加data "alicloud_alikafka_topics" "topics_ds" 配置项,具体配置如下。

    data "alicloud_alikafka_topics" "topics_ds" {
      instance_id = alicloud_alikafka_instance.default.id
    }
  2. 创建执行计划,并预览变更。

    terraform plan
  3. 执行以下命令,创建资源。

    terraform apply

    在执行过程中,根据提示输入yes并按下Enter键,等待命令执行完成,若出现以下信息,则表示运行成功。

    No changes. Your infrastructure matches the configuration.
    
    Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
    
    Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
    
    Outputs:
    
    first_topic_name = "testtopic"
  4. 验证结果。执行terraform show查看Kafka Topic信息。

    terraform show
    # data.alicloud_alikafka_topics.topics_ds:
    data "alicloud_alikafka_topics" "topics_ds" {
        id          = "29****"
        ids         = [
            "alikafka_post-cn-****:example-topic",
        ]
        instance_id = "alikafka_post-cn-****"
        names       = [
            null,
        ]
        output_file = "topics.txt"
        page_size   = 50
        topics      = [
            {
                compact_topic = false
                create_time   = "1732****"
                id            = "alikafka_post-cn-****:example-topic"
                instance_id   = "alikafka_post-cn-****"
                local_topic   = false
                partition_num = 24
                remark        = "updated_kafka_topic_remark"
                status        = 0
                status_name   = "服务中"
                tags          = {}
                topic         = "example-topic"
            },
        ]
        total_count = 1
    }

清理资源

当您不再需要上述通过Terraform创建或管理的资源时,请运行以下命令以释放资源。关于terraform destroy的更多信息,请参见Terraform常用命令

terraform destroy

完整示例

说明

当前示例代码支持一键运行,您可以直接运行代码。一键运行

示例代码

variable "region" {
  default = "cn-shenzhen"
}

variable "instance_name" {
  default = "alikafkaInstanceName"
}

variable "zone_id" {
  default = "cn-shenzhen-f"
}

variable "partition_num" {
  default = "12"
}

variable "remark" {
  default = "kafka_topic_remark"
}

provider "alicloud" {
  region = var.region
}

# 创建VPC
resource "alicloud_vpc" "default" {
  vpc_name   = "alicloud"
  cidr_block = "172.16.0.0/16"
}

# 创建交换机
resource "alicloud_vswitch" "default" {
  vpc_id     = alicloud_vpc.default.id
  cidr_block = "172.16.192.0/20"
  zone_id    = var.zone_id
}

# 创建安全组。
resource "alicloud_security_group" "default" {
  vpc_id = alicloud_vpc.default.id
}

# 创建实例,磁盘类型为高效云盘,磁盘容量为500 GB,流量规格为alikafka.hw.2xlarge。
# 部署实例。
resource "alicloud_alikafka_instance" "default" {
  name           = var.instance_name
  partition_num  = 50
  disk_type      = 0
  disk_size      = 500
  deploy_type    = 5
  io_max_spec    = "alikafka.hw.2xlarge"
  vswitch_id     = alicloud_vswitch.default.id
  security_group = alicloud_security_group.default.id
}

# 创建Kafka Topic
resource "alicloud_alikafka_topic" "default" {
  instance_id   = alicloud_alikafka_instance.default.id
  topic         = "example-topic"
  local_topic   = "false"
  compact_topic = "false"
  partition_num = var.partition_num
  remark        = var.remark
}

# 查询Kafka Topic
data "alicloud_alikafka_topics" "topics_ds" {
  instance_id = alicloud_alikafka_instance.default.id
  depends_on = [alicloud_alikafka_topic.default]
}

相关文档